Multiple data channel memory module architecture

ABSTRACT

According to one example of the present disclosure, a system includes a computing element configured to provide requests for memory access operations and a memory module comprising a plurality of memories, a plurality of independent data channels, each of the independent data channels coupled to one of the plurality of memories, a plurality of internal address/control channels, each of the independent address/control channels coupled to one of the plurality of memories, and control logic coupled to the plurality of internal address/control channels and configured to receive and decode address and control information for a memory access operation, the control logic further configured to selectively provide the decoded address and control information to a selected internal address/control channel for a selected independent data channel of the plurality of independent data channels based on the received address and control information for the memory access operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of patent application Ser. No.15/806,217 filed Nov. 7, 2017, issued as U.S. Pat. No. 10,061,699 onAug. 28, 2018, which is a continuation of U.S. patent application Ser.No. 15/254,975 filed Sep. 1, 2016, issued as U.S. Pat. No. 9,824,010 onNov. 21, 2017, which is a continuation of U.S. patent application Ser.No. 14/673,732, filed Mar. 30, 2015, issued as U.S. Pat. No. 9,449,659,on Sep. 20, 2016, which is a continuation of U.S. patent applicationSer. No. 12/186,372, filed Aug. 5, 2008, issued as U.S. Pat. No.9,015,399 on Apr. 21, 2015. The present application also relates to thefollowing commonly-assigned U.S. Patent Applications: 1) U.S. patentapplication Ser. No. 11/841,406, filed Aug. 20, 2007 titled“MULTI-PROCESSOR SYSTEM HAVING AT LEAST ONE PROCESSOR THAT COMPRISES ADYNAMICALLY RECONFIGURABLE INSTRUCTION SET”, issued as U.S. Pat. No.8,156,307 on Apr. 10, 2012, 2) U.S. patent application Ser. No.11/854,432, filed Sep. 12, 2007 titled “DISPATCH MECHANISM FORDISPATCHING INSTRUCTIONS FROM A HOST PROCESSOR TO A CO-PROCESSOR”,issuing as U.S. Pat. No. 8,122,229 on Feb. 21, 2012, 3) U.S. patentapplication Ser. No. 11/847,169, filed. Aug. 29, 2007 titled “COMPILERFOR GENERATING AN EXECUTABLE COMPRISING INSTRUCTIONS FOR A PLURALITY OFDIFFERENT INSTRUCTION SETS”, issued as U.S. Pat. No. 8,561,037 on Oct.15, 2013, 4) U.S. patent application Ser. No. 11/969,792, filed Jan. 4,2008 titled “MICROPROCESSOR ARCHITECTURE HAVING ALTERNATIVE MEMORYACCESS PATHS”, issued as U.S. Pat. No. 9,710,384 on Jul. 18, 2017, and5) U.S. patent application Ser. No. 12/186,344, filed Aug. 5, 2008titled “MEMORY INTERLEAVE FOR HETEROGENEOUS COMPUTING”, issued as U.S.Pat. No. 8,095,735 on Jan. 10, 2012. These applications and patents areincorporated by reference, herein, in their entirety and for allpurposes.

TECHNICAL FIELD

The following description relates generally to memory modules, such asdual in-line memory modules (DIMMs), and more particularly to a memorymodule architecture that has multiple data channels (i.e., a multi-datachannel memory module architecture). In certain embodiments, a memorymodule comprises a plurality of data channels that each enable asub-cache-block of data to be accessed for independent operations.Further, in certain embodiments, multiple ones of the data channels maybe employed to support a cache-block access of data.

BACKGROUND

The popularity of computing systems continues to grow and the demand forimproved processing architectures thus likewise continues to grow.Ever-increasing desires for improved computing performance/efficiencyhas led to various improved processor architectures. For example,multi-core processors are becoming more prevalent in the computingindustry and are being used in various computing devices, such asservers, personal computers (PCs), laptop computers, personal digitalassistants (PDAs), wireless telephones, and so on.

In the past, processors such as CPUs (central processing units) featureda single execution unit to process instructions of a program. Morerecently, computer systems are being developed with multiple processorsin an attempt to improve the computing performance of the system. Insome instances, multiple independent processors may be implemented in asystem. In other instances, a multi-core architecture may be employed,in which multiple processor cores are amassed on a single integratedsilicon die. Each of the multiple processors (e.g., processor cores) cansimultaneously execute program instructions. This parallel operation ofthe multiple processors can improve performance of a variety ofapplications.

A multi-core CPU combines two or more independent cores into a singlepackage comprised of a single piece silicon integrated circuit (IC),called a die. In some instances, a multi-core CPU may comprise two ormore dies packaged together. A dual-core device contains two independentmicroprocessors and a quad-core device contains four microprocessors.Cores in a multi-core device may share a single coherent cache at thehighest on-device cache level (e.g., L2 for the Intel® Core 2) or mayhave separate caches (e.g. current AMD® dual-core processors). Theprocessors also share the same interconnect to the rest of the system.Each “core” may independently implement optimizations such assuperscalar execution, pipelining, and multithreading. A system with Ncores is typically most effective when it is presented with N or morethreads concurrently.

One processor architecture that has been developed utilizes multipleprocessors (e.g., multiple cores), which are homogeneous. As discussedhereafter, the processors are homogeneous in that they are allimplemented with the same fixed instruction sets (e.g., Intel's x86instruction set, AMD's Opteron instruction set, etc.). Further, thehomogeneous processors access memory in a common way, such as all of theprocessors being cache-line oriented such that they access a cache block(or “cache line”) of memory at a time, as discussed further below.

In general, a processor's instruction set refers to a list of allinstructions, and all their variations, that the processor can execute.Such instructions may include, as examples, arithmetic instructions,such as ADD and SUBTRACT; logic instructions, such as AND, OR, and NOT;data instructions, such as MOVE, INPUT, OUTPUT, LOAD, and STORE; andcontrol flow instructions, such as GOTO, if X then GOTO, CALL, andRETURN. Examples of well-known instruction sets include x86 (also knownas IA-32), x86-64 (also known as AMD64 and Intel® 64), AMD's Opteron,VAX (Digital Equipment Corporation), IA-64 (Itanium), and PA-RISC (HPPrecision Architecture).

Generally, the instruction set architecture is distinguished from themicroarchitecture, which is the set of processor design techniques usedto implement the instruction set. Computers with differentmicroarchitectures can share a common instruction set. For example, theIntel® Pentium and the AMD® Athlon implement nearly identical versionsof the x86 instruction set, but have radically different internalmicroarchitecture designs. In all these cases the instruction set (e.g.,x86) is fixed by the manufacturer and directly hardware implemented, ina semiconductor technology, by the microarchitecture. Consequently, theinstruction set is traditionally fixed for the lifetime of thisimplementation.

FIG. 1 shows a block-diagram representation of an exemplary prior artsystem 100 in which multiple homogeneous processors (or cores) areimplemented. System 100 comprises two subsystems: 1) a main memory(physical memory) subsystem 101 and 2) a processing subsystem 102 (e.g.,a multi-core die). System 100 includes a first microprocessor core 104Aand a second microprocessor core 104B. In this example, microprocessorcores 104A and 104B are homogeneous in that they are each implemented tohave the same, fixed instruction set, such as x86. In addition, each ofthe homogeneous microprocessor cores 104A and 104B access main memory101 in a common way, such as via cache block accesses, as discussedhereafter. Further, in this example, cores 104A and 104B are implementedon a common die 102. Main memory 101 is communicatively connected toprocessing subsystem 102. Main memory 101 comprises a common physicaladdress space that microprocessor cores 104A and 104B can eachreference.

As shown further in FIG. 1, a cache 103 is also implemented on die 102.Cores 104A and 104B are each communicatively coupled to cache 103. As iswell known, a cache generally is memory for storing a collection of dataduplicating original values stored elsewhere (e.g., to main memory 101)or computed earlier, where the original data is expensive to fetch (dueto longer access time) or to compute, compared to the cost of readingthe cache. In other words, a cache 103 generally provides a temporarystorage area where frequently accessed data can be stored for rapidaccess. Once the data is stored in cache 103, future use can be made byaccessing the cached copy rather than re-fetching the original data frommain memory 101, so that the average access time is shorter. In manysystems, cache access times are approximately 50 times faster thansimilar accesses to main memory 101. Cache 103, therefore, helpsexpedite data access that the micro-cores 104A and 104B would otherwisehave to fetch from main memory 101.

In many system architectures, each core 104A and 104B will have its owncache also, commonly called the “L1” cache, and cache 103 is commonlyreferred to as the “L2” cache. Unless expressly stated herein, cache 103generally refers to any level of cache that may be implemented, and thusmay encompass L1, L2, etc. Accordingly, while shown for ease ofillustration as a single block that is accessed by both of cores 104Aand 104B, cache 103 may include L1 cache that is implemented for eachcore.

In many system architectures, virtual addresses are utilized. Ingeneral, a virtual address is an address identifying a virtual(non-physical) entity. As is well-known in the art, virtual addressesmay be utilized for accessing memory. Virtual memory is a mechanism thatpermits data that is located on a persistent storage medium (e.g., disk)to be referenced as if the data was located in physical memory.Translation tables, maintained by the operating system, are used todetermine the location of the reference data (e.g., disk or mainmemory). Program instructions being executed by a processor may refer toa virtual memory address, which is translated into a physical address.To minimize the performance penalty of address translation, most modernCPUs include an on-chip Memory Management Unit (MMU), and maintain atable of recently used virtual-to-physical translations, called aTranslation Look-aside Buffer (TLB). Addresses with entries in the TLBrequire no additional memory references (and therefore time) totranslate. However, the TLB can only maintain a fixed number of mappingsbetween virtual and physical addresses; when the needed translation isnot resident in the TLB, action will have to be taken to load it in.

As an example, suppose a program's instruction stream that is beingexecuted by a processor, say processor core 104A of FIG. 1, desires toload data from an address “Foo” into a first general-purpose register,GPR1. Such instruction may appear similar to “LD<Foo>, GPR1”. Foo, inthis example, is a virtual address that the processor translates to aphysical address, such as address “123456”. Thus, the actual physicaladdress, which may be formatted according to a global physical memoryaddress format, is used to access cache 103 and/or memory 101.

In operation, each of cores 104A and 104B reference main memory 101 byproviding a physical memory address. The physical memory address (ofdata or “an operand” that is desired to be retrieved) is first presentedto cache 103. If the addressed data is not encached (i.e., not presentin cache 103), the same physical address is presented to main memory 101to retrieve the desired data. Main memory 101 may be implemented inwhole or in part via memory module(s), such as dual in-line memorymodules (DIMMs), which may employ dynamic random access memory (DRAM) orother memory storage.

In contemporary architectures, the processor cores 104A and 104B arecache-line (or “cache-block”) oriented, wherein a “cache block” isfetched from main memory 101 and loaded into cache 103. The terms cacheline and cache block are used interchangeably herein. Rather thanretrieving only the addressed data from main memory 101 for storage tocache 103, such cache-block oriented processors may retrieve a largerblock of data for storage to cache 103. A cache block typicallycomprises a fixed-size amount of data that is independent of the actualsize of the requested data. For example, in most implementations a cacheblock comprises 64 bytes of data that is fetched from main memory 101and loaded into cache 103 independent of the actual size of the operandreferenced by the requesting micro-core 104A/104B. Furthermore, thephysical address of the cache block referenced and loaded is a blockaddress. This means that all the cache block data is in sequentiallycontiguous physical memory. Table 1 below shows an example of a cacheblock.

TABLE 1 Physical Address Operand XXX(7) Operand 7 XXX(6) Operand 6 . . .. . . XXX(1) Operand 1 XXX(0) Operand 0

In the above example of table 1, the “XXX” portion of the physicaladdress is intended to refer generically to the corresponding identifier(e.g., numbers and/or letters) for identifying a cache line address. Forinstance, XXX(0) corresponds to the physical address for an Operand 0,while XXX(1) corresponds to the physical address for an Operand 1, andso on. In the example of table 1, in response to a micro-core 104A/104Brequesting Operand 0 via its corresponding physical address XXX(0), a64-byte block of data may be fetched from main memory 101 and loadedinto cache 103, wherein such cache block of data includes not onlyOperand 0 but also Operands 1-7. Thus, depending on the fixed size ofthe cache block employed on a given system, whenever a core 104A/104Breferences one operand (e.g., a simple load), the memory system willbring in 4 to 8 to 16 (or more) operands into cache 103.

There are both advantages and disadvantages of this traditionalcache-block oriented approach to memory access. One advantage is that ifthere is temporal (over time) and spatial (data locality) references tooperands (e.g., operands 0-7 in the example of Table 1), then cache 103reduces the memory access time. Typically, cache access times (and databandwidth) are 50 times faster than similar access to main memory 101.For many applications, this is the memory access pattern.

However, if the memory access pattern of an application is notsequential and/or does not re-use data, inefficiencies arise whichresult in decreased performance. Consider the following FORTRAN loopthat may be executed for a given application:

-   -   DO I=1, N, 4    -   A(i)=B(i)+C(i)    -   END DO

In this loop, every fourth element is used. If a cache block maintains 8operands, then only 2 of the 8 operands are used. Thus, 6/8 of the dataloaded into cache 103 and 6/8 of the memory bandwidth is “wasted” inthis example.

In multi-processor systems, such as exemplary system 100 of FIG. 1, mainmemory 101 can be configured to improve performance. FIG. 2 shows ablock diagram illustrating a traditional implementation of main memory101. As shown, memory module 202, which comprises memory (e.g., DRAMs)203, is accessible via memory controller 201. That is, memory controller201 controls access to memory module 202. Memory module 202 is commonlyimplemented as a DIMM (dual in-line memory module) that includes one ormore DRAMs (dynamic random access memory) as memory 203. In general, aDIMM is a double SIMM (single in-line memory module). Like a SIMM, aDIMM contains one or several random access memory (RAM) chips on a smallcircuit board with pins that connect it to the computer motherboard.

Traditional DIMMs provide one data channel 205 and one address/controlchannel 204 per DIMM. In general, the address/control channel 204specifies an address and a desired type of access (e.g., read or write),and the data channel 205 carries the corresponding data to/from thespecified address for performing the desired type of access. Typically,a memory access operation requires several clock cycles to perform. Forinstance, address and control information may be provided on theaddress/control channel 204 over one or more clock cycles, and then thedata is provided on the data channel 205 over later clock cycles. In atypical DIMM access scenario, a row select command is sent from memorycontroller 201 on the address/control channel 204 to the memory module202, which indicates that an associated address is a row address in thememory cell matrix of the DRAM memory 203. In general, a data bit inDRAM is stored in a memory cell located by the intersection of a columnaddress and a row address. A column access command (e.g., a column reador column write command) is sent from the memory controller 201 over theaddress/control channel 204 to validate the column address and indicatea type of access desired (e.g., either a read or write operation).

The row select command may be sent in a first clock cycle, then thecolumn access command may be sent in a second clock cycle, and then someclock cycles later a burst of data may be supplied via the data channel204. The burst of data may be supplied over several clock cycles.Typically, single DIMM data channel 205 is typically a 64-bit (8-byte)wide channel, wherein each access comprises a “burst” length of 8, thusresulting in the data channel carrying 64 bytes for each access. Thelength of the “burst” may refer to a number of clock cycles or phases ofa clock cycle when dual-data rate (DDR) is employed. For instance, aburst length of 8 may refer to 8 clock cycles, wherein 8 bytes of datais communicated on the data channel for a given access in each of the 8clock cycles (resulting in the data channel carrying 64 bytes of datafor the access). As another example, a burst length of 8 may refer to 8phases of a clock (e.g., when DDR is employed), wherein 8 bytes of datais communicated on the data channel for a given access in each of the 8phases (over 4 clock cycles), thus resulting in the data channelcarrying 64 bytes of data for the access.

To improve data channel bandwidth, tiling is commonly employed in memoryarchitectures. For instance, rather than waiting for completion of aburst of data for one access operation before supplying address/controlsignals for a next access operation, the instructions supplied via theaddress/control channel 204 may be used to attempt to maintain fullbandwidth utilization of the data channel 205. FIG. 3 shows an exampleof one traditional tiling technique. FIG. 3 shows a clock cycle 301 of areference clock signal, wherein the illustrated example shows 20 clockcycles numbered 1-20. A clock phase 302 is also shown, wherein for eachclock cycle the clock has a low phase (“L”) and a high phase (“H”), asis well known. An address/control channel 303 is also shown, whichcorresponds to address/control channel 204 of FIG. 2. Also, in thisexample, a data channel 304 is shown, which corresponds to data channel205 of FIG. 2.

The exemplary tiling technique of FIG. 3 allows for the address/controlchannel 303 to be used to maintain high bandwidth utilization of thedata channel 304. In the illustrated example, a first memory accessoperation is requested, whereupon a row select command 306 iscommunicated from memory controller 201 to memory module 202 overaddress/control channel 303 during clock cycle 1. Then, during clockcycle 2, a column access command (e.g., column read or column writecommand) 307 for the first memory access operation is communicated frommemory controller 201 to memory module 202 over address/control channel303. After some delay, data channel 304 carries the data “burst” for thefirst memory access operation. For instance, beginning in the high phaseof clock cycle 9 and ending in the low phase of clock cycle 13, databurst 308 carries the data for the first memory access operation.Traditionally, a single DIMM data channel, such as data channel 304, istypically a 64-bit (8-byte) wide channel where each memory accesscomprises a “burst” length of 8, thus resulting in the data channelcarrying 64 bytes for each access. For instance, each of the 8 blocks ofburst 308 (labeled 0/0/0-0/0/7) is typically an 8-byte block of data,thus resulting in burst 308 containing 64 bytes of data for the firstmemory access operation (read or write to/from the specified address).

A second memory access operation is requested in this example, whereupona row select command 309 is communicated from memory controller 201 tomemory module 202 over address/control channel 303 during clock cycle 5.Then, during clock cycle 6, a column access command 310 for the secondmemory access operation is communicated from memory controller 201 tomemory module 202 over address/control channel 303. After some delay,data channel 304 carries the data “burst” for the second memory accessoperation. For instance, beginning in the high phase of clock cycle 13and ending in the low phase of clock cycle 17, data burst 311 carriesthe data for the second memory access operation. As with the data burst308 discussed above for the first memory access operation, data burst311 typically has a length of 8 blocks (labeled 0/1/0-0/1/7) that areeach an 8-byte block of data, thus resulting in burst 311 containing 64bytes of data for the third memory access operation (read or writeto/from the specified address).

As the example of FIG. 3 illustrates, rather than waiting for the databurst 308 for a first memory access operation to complete beforeproviding the address/control information for the next memory accessoperation to be performed, the tiling technique uses the address/controlchannel 303 to effectively schedule the data bursts for different memoryaccess operations back-to-back, thereby maintaining high bandwidthutilization on the data channel 304.

As also illustrated in FIG. 3, traditionally the data channel 205 of aDIMM carries a 64-byte burst of data for each memory access operationrequested. Some DIMMs can support 64-byte or 32-byte accesses. That is,some DIMMs may be configured into either a 64-byte access or a 32-byteaccess mode. Thus, memory bandwidth may be conserved to some extent forcertain memory access operations by performing a 32-byte access of theDIMM, rather than a 64-byte access (if the operation only requiresaccess of 32 or fewer bytes). However, the full burst of either 32-bytesor 64-bytes is utilized for a single memory access operation.

In certain implementations, a plurality of DIMMs may share anaddress/control channel, and each DIMM may provide a separate datachannel, wherein tiling may be employed on the address/control channelto maintain high bandwidth utilization on both data channels of theDIMMs. However, in these implementations, each DIMM provides only asingle data channel.

As is well-known in the art, memory is often arranged into independentlycontrollable arrays, often referred to as “memory banks.” Under thecontrol of a memory controller, a bank can generally operate on onetransaction at a time. As mentioned above, the memory may be implementedby dynamic storage technology (such as “DRAMS”), or of static RAMtechnology. In a typical DRAM chip, some number (e.g., 4, 8, andpossibly 16) of banks of memory may be present. A memory interleavingscheme may be desired to minimize one of the banks of memory from beinga “hot spot” of the memory.

In most systems, memory 101 may hold both programs and data. Each hasunique characteristics pertinent to memory performance. For example,when a program is being executed, memory traffic is typicallycharacterized as a series of sequential reads. On the other hand, when adata structure is being accessed, memory traffic is usuallycharacterized by a stride, i.e., the difference in address from aprevious access. A stride may be random or fixed. For example,repeatedly accessing a data element in an array may result in a fixedstride of two. As is well-known in the art, a lot of algorithms have apower of 2 stride. This power of 2 stride gives rise to an increase inoccurrences of bank conflicts because the power of 2 stride ends upaccessing the same bank repeatedly. Accordingly, without some memoryinterleave management scheme being employed, hot spots may beencountered within the memory in which a common portion of memory (e.g.,a given bank of memory) is accessed much more often than other portionsof memory.

As discussed above, many compute devices, such as the Intel x86 or AMDx86 microprocessors, are cache-block oriented. Today, a cache block of64 bytes in size is typical, but compute devices may be implemented withother cache block sizes. A cache block is typically contained all on asingle hardware memory storage element, such as a single dual in-linememory module (DIMM). As discussed above, when the cache-block orientedcompute device accesses that DIMM, it presents one address and isreturned the entire cache-block (e.g., 64 bytes), as in the exemplarydata bursts 308 and 311 discussed above with FIG. 3.

Some compute devices, such as certain accelerator compute devices, maynot be cache-block oriented. That is, those non-cache-block orientedcompute devices may access portions of memory (e.g., words) on a muchsmaller, finer granularity than is accessed by the cache-block orientedcompute devices. For instance, while a typical cache-block orientedcompute device may access a cache block of 64 bytes for a single memoryaccess request, a non-cache-block oriented compute device may desire toaccess a Word that is 8 bytes in size in a single memory access request.That is, the non-cache-block oriented compute device in this example maydesire to access a particular memory DIMM and only obtain 8 bytes from aparticular address present in the DIMM.

As discussed above, traditional multi-processor systems have employedhomogeneous compute devices (e.g., processor cores 104A and 104B ofFIG. 1) that each access memory 101 in a common manner, such as viacache-block oriented accesses. While some systems may further includecertain heterogeneous compute elements, such as accelerators (e.g., aGPU), the heterogeneous compute element does not share the same physicalor virtual address space of the homogeneous compute elements.Accordingly, traditional memory interleave schemes have not attempted toaddress an interleave of memory accesses across heterogeneous computeelements, which may access memory in different ways, such as viacache-block and non-cache-block accesses.

U.S. Patent Application Publication No. 2007/0266206 to Kim et al.(hereinafter “Kim”) proposes a scatter-gather intelligent memoryarchitecture. Kim mentions that to avoid wasting memory bandwidth, thescatter/gather engine supports both cache line size data accesses andsmaller, sub-cache line accesses. However, Kim does not appear todescribe its memory architecture in detail. One of ordinary skill in theart would thus suppose that Kim may be employing the above-mentionedtraditional DIMMs, which enable either a full cache line (e.g., 64bytes) or a sub-cache line (e.g., 32 bytes) access. However, as with thetraditional DIMMs, only a single data channel per DIMM appears to besupported. Kim does not appear to provide any disclosure of a DIMMarchitecture that provides more than a single data channel per DIMM.

SUMMARY

The present invention is directed generally to systems and methods whichprovide a memory module having multiple data channels that areindependently accessible (i.e., a multi-data channel memory module).According to one embodiment, the multi-data channel memory moduleenables a plurality of independent sub-cache-block accesses to beserviced simultaneously. In addition, the memory architecture alsosupports cache-block accesses. For instance, multiple ones of the datachannels may be employed for servicing a cache-block access. In certainembodiments, the memory module is a scatter/gather dual in-line memorymodule (DIMM).

Thus, in one embodiment a DIMM architecture that comprises multiple datachannels is provided. Each data channel supports a sub-cache-blockaccess, and multiple ones of the data channels may be used forsupporting a cache-block access. The plurality of data channels to agiven DIMM may be used simultaneously to support different, independentoperations (or access requests).

According to one exemplary embodiment, a memory module (e.g., DIMM)comprises eight 8-byte data access channels. Thus, eight 8-byte accessescan be performed in parallel on the given memory module. As an example,a first of the access channels may be performing a read access of asub-cache-block of data, while another of the access channels may besimultaneously performing a write access of a sub-cache-block of data.

Thus, instead of having a single 64-byte access bus (or data channel)for the memory module, as with traditional DIMMs, in certain embodimentsthe access bus (or data channel) is partitioned into 8 independent8-byte sub-buses (which may also be referred to as channels, paths, orlanes). An address and a request type is independently supported foreach of the 8-byte sub-buses individually. Accordingly, in certainembodiments, one may think of the traditional DIMM data channel as beingdivided into multiple sub-buses, which may be referred to as data pathsor lanes. Of course, because each of these sub-buses are independentlyaccessible (e.g., for supporting independent memory access operations),they are similar to separate data channels, rather than being smallerportions (e.g., “lanes”) of a larger overall data channel. As such, thesub-buses may be referred to herein as separate data channels, datalanes, or data paths, and each of these terms is intended to have thesame meaning, effectively providing for multiple, independentlyaccessible data channels (which may each support a sub-cache-blockaccess of data) for a memory module.

As discussed further hereafter, the 8 independent sub-buses may be usedto simultaneously support different sub-cache-block accesses.Additionally, multiple ones of the independent sub-buses may be employedto satisfy a cache-block access. For instance, the eight 8-bytesub-buses may be used to satisfy a full 64-byte cache-block access. Asfurther discussed hereafter, in certain embodiments the cache-block andsub-cache-block accesses may be intermingled such that all eight of the8-byte data channels need not be reserved for simultaneous use insatisfying a cache-block access. Rather, in certain embodiments, thecache-block access may be satisfied by the channels within a window oftime, wherein logic (e.g., a memory controller) may receive thecache-block data within the window of time and bundle the received datainto a cache-block of data for satisfying a cache-block access request.

According to one embodiment, the traditional 64-byte data channel of aDIMM (such as the exemplary data channel 205 discussed above with FIG.2) is partitioned into a plurality of “lanes” such that the bandwidth ofthe traditional data channel is leveraged to support multipleindependent sub-cache-block accesses. For instance, in one embodiment,the 64-bit wide data path of a traditional DIMM is partitioned intoeight 8-bit wide paths that each have independent control. As anexample, in one embodiment, each lane is 8 bits (1-byte) wide, whereineach data access comprises a “burst” length of 8, thus resulting in eachdata lane carrying 8 bytes for each access. Thus, the traditional64-byte data channel of a DIMM (such as that discussed above in FIG. 2)is, in one embodiment, effectively divided into 8 independent data lanesthat each support an independent access of 8 bytes. Accordingly, whereasa traditional data channel of FIG. 2 provides a 64-byte data burst for agiven memory access operation (read or write), a DIMM according to oneembodiment effectively sub-divides the traditional DIMM data channel toprovide eight independent 8-byte data bursts for potentially supportingeight independent 8-byte access operations (read or write operations).

In one embodiment, when a sub-cache-block access (e.g., a single word)is requested, the address of the sub-cache-block to be accessed issupplied to one of the eight sub-buses (or data lanes) with acorresponding request type (e.g., read or write), and that sub-busprovides the sub-cache-block of data. The other seven 8-byte sub-busescan each independently be supporting other operations. On the otherhand, when a cache block access (e.g., of 64-bytes) is requested, thesame address and request type (e.g., either a read or write) may besupplied to all eight sub-buses. The eight sub-buses each returns theirrespective portion of the requested cache block so that the entire cacheblock is returned in a single burst by the eight sub-buses.

In certain embodiments, upon receiving a cache-block access request, theeight sub-buses may be reserved (to place any sub-cache-block accessrequests received thereafter “on hold” until the eight sub-buses areused for satisfying the cache-block access request), and the eightsub-buses may then be used simultaneously to fully, in one burst,satisfy the 64-byte cache-block access request. As discussed furtherhereafter, in other embodiments, no such reservation is employed, butinstead the cache-block access request may be handled by the eightsub-buses along with an intermingling of any sub-cache-block accessrequests that might be present at that time, wherein the cache-blockaccess may be satisfied by the sub-buses within a window of time, andthe 64 bytes of the cache-block access returned by the sub-buses withinthe window of time may be bundled by logic (e.g., a memory controller)into the requested 64 byte cache block of data. Thus, rather thansupplying the same address and request type (e.g., either a read orwrite) to all eight sub-buses simultaneously for satisfying acache-block access request, in certain embodiments, such address andrequest type for the cache-block access may in a first instance besupplied to a portion of the eight sub-buses (which each returns theirrespective portion of the requested cache block) and in a later instancea further portion of the eight sub-buses may be supplied the address andrequest type in order to return the remaining portion of the requestedcache block. The two portions of the cache block may then bundledtogether (e.g., by a memory controller) to form the requested cacheblock of data. In other words, rather than satisfying a cache-blockaccess in a single burst of data, in certain embodiments portions of thecache-block of data may be returned over a plurality of bursts (e.g.,with sub-cache-block bursts of data intermingled therewith), and theappropriate portions may be bundled together to form a congruent burstof cache-block data.

Thus, in certain embodiments, cache-block (e.g., 64-byte) accesses maybe intermixed with sub-cache-block (e.g., 8-byte) accesses, and each8-byte sub-bus (or “lane”) of the memory module is scheduledindependently to support the intermixing. Thus, a cache-block access maynot necessarily be performed using all eight sub-buses simultaneously(such that the entire cache-block is returned in a single burst in themanner mentioned above), but instead, at a given time some of the eight8-byte sub-buses may be used for performing a sub-cache-block accesswhile some others of the eight 8-byte sub-buses are used for thecache-block access. Thus, the cache-block access may be returned withina window of time by the sub-buses, wherein a controller bundles thereturned data into the requested cache-block.

In one embodiment, the memory module comprises control logic, such as aField-Programmable Gate Array (FPGA), that manages decoding andmultiplexing of address and control information for the plurality ofdata channels of the module. For instance, in certain embodiments,address and control information for memory access operations iscommunicated from a memory controller to the memory module via anexternal address/control channel. In certain embodiments, the addressand control information is encoded according to a time multiplexedencoding scheme to enable address and control information for aplurality of independent memory access operations to be received over acommunication time period (e.g., over two time units) in which addressand control information for a single memory access operation istraditionally communicated. For instance, during the communication timeperiod that is traditionally performed on an address/control channel forspecifying the address and control information for a 64-byte memoryaccess operation (e.g., read or write), the encoded address/controlchannel of certain embodiments carries information specifying theaddress and control information for a plurality of independentsub-cache-block data access operations (e.g., eight 8-byte data accessoperations).

The control logic receives the encoded address and control informationand decodes that information to control the plurality of data channelsfor servicing the plurality of memory access operations specified in thereceived encoded address and control information. In certainembodiments, a plurality of internal address/control channels isemployed within the memory module, which are used for controlling theplurality of data channels for servicing a plurality of independentmemory access operations, as discussed further herein.

According to certain embodiments of the present invention, rather thanservicing a single memory access operation over a traditional singlememory access time period (e.g., an 8 time unit burst), multiple datachannels are employed in a memory module (e.g., DIMM) to service aplurality of independent memory access operations over the same accesstime period. For instance, rather than carrying 64-bytes of data for asingle memory access operation over an 8 time unit burst (e.g., 8 clockunits or 8 clock phases), an embodiment of the multi-data channel memorymodule disclosed herein carries 8-bytes of data for each of a pluralityof independent memory access operations over such an 8 time unit burst.Thus, according to one embodiment, over an access time period forcarrying a cache-block of data (e.g., an 8 time unit burst of 64-bytesof data), the multi-data channel memory module carries a sub-cache-blockof data for each of a plurality of independent memory access operations(e.g., carries 8-bytes of data for each of eight independent memoryaccess operations).

Some computing systems are being developed that include heterogeneouscompute elements that share a common physical and/or virtual addressspace of memory. As an example, a system may comprise one or morecompute elements that are cache-block oriented, and the system mayfurther comprise one or more compute elements that are non-cache-blockoriented. For instance, the cache-block oriented compute element(s) mayaccess main memory in cache blocks of, say, 64 bytes per request,whereas the non-cache-block oriented compute element(s) may access mainmemory via smaller-sized requests (which may be referred to as“sub-cache-block” requests), such as 8 bytes per request.

One exemplary heterogeneous computing system that may include one ormore cache-block oriented compute elements and one or morenon-cache-block oriented compute elements is that disclosed inco-pending U.S. patent application Ser. No. 11/841,406 filed Aug. 20,2007 titled “MULTI-PROCESSOR SYSTEM HAVING AT LEAST ONE PROCESSOR THATCOMPRISES A DYNAMICALLY RECONFIGURABLE INSTRUCTION SET”, the disclosureof which is incorporated herein by reference. For instance, in such aheterogeneous computing system, one or more processors may becache-block oriented, while one or more other processors (e.g., theprocessor described as comprising a dynamically reconfigurableinstruction set) may be non-cache-block oriented, and the heterogeneousprocessors share access to the common main memory (and share a commonphysical and virtual address space of the memory).

Accordingly, a desire has arisen for an efficient memory architecturefor supporting differently sized memory access requests, such as theabove-mentioned cache-block accesses and sub-cache-block accesses. Suchan improved memory architecture is desired, for example, for use incomputing systems that may include one or more cache-block orientedcompute elements and one or more non-cache-block oriented computeelements. While the exemplary heterogeneous computing system disclosedin U.S. patent application Ser. No. 11/841,406 filed Aug. 20, 2007titled “MULTI-PROCESSOR SYSTEM HAVING AT LEAST ONE PROCESSOR THATCOMPRISES A DYNAMICALLY RECONFIGURABLE INSTRUCTION SET” is one exampleof a system for which an improved memory architecture may be desired,embodiments of the improved multi-data channel memory modulearchitecture described herein are not limited for use with thatheterogeneous computing system, but may likewise be applied to variousother types of heterogeneous computing systems in which cache-blockoriented and non-cache-block oriented compute elements (e.g.,processors) share access to a common memory. In addition, embodimentsmay likewise be used within homogeneous computing systems.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawing, in which:

FIG. 1 shows a block diagram of an exemplary system architecture of theprior art;

FIG. 2 shows a block diagram of an exemplary memory system of the priorart;

FIG. 3 shows an exemplary tiling of memory access operations of theprior art;

FIG. 4 shows an exemplary system according to one embodiment of thepresent invention;

FIG. 5 shows a block diagram of an exemplary memory system according toone embodiment of the present invention;

FIG. 6 shows an exemplary simplified schematic of a DIMM implemented inaccordance with one embodiment of the present invention;

FIG. 7 shows an exemplary tiling of memory access operations accordingto one embodiment of the present invention; and

FIG. 8 shows an exemplary system in which multi-data channel memorymodules are implemented according to one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Turning to FIG. 4, a block diagram of an exemplary system 40 accordingto one embodiment of the present invention is shown. System 40 comprisescompute element(s) 41, memory controller 42, and a memory module (e.g.,DIMM) 43. Memory module 43 comprises multiple, independently controlleddata channels (data channels 0-N), and may thus be referred to as amulti-data channel memory module. As discussed above, the multiple,independently controlled data channels may, in some embodiments, beformed by partitioning the single data channel of a traditional DIMMinto multiple, independently controlled data channels. For instance,instead of having a single 64-byte data channel for the memory module,as with traditional DIMMs, in certain embodiments the data channel ispartitioned into eight independent 8-byte data channels (which may alsobe referred to as paths or lanes). An address and a request type isindependently supported for each of the 8-byte data channels 0-Nindividually. Thus, as discussed further hereafter, the independent datachannels 0-N may be used to support independent sub-cache-blockaccesses.

The combination of elements 41-43 permit programs to be executed, i.e.instructions are executed in compute element(s) 41 to process datastored in memory 402 of memory module 403. Compute element(s) 41 may beprocessors (e.g., processor cores) or other functional units. Computeelement(s) 41 may comprise a plurality of compute elements, such asprocessor cores 104A and 104B of FIG. 1. In certain embodiments, suchcompute element(s) 41 comprise heterogeneous compute elements, asdiscussed further herein. For instance, such compute elements may beheterogeneous in that they access memory 402 of memory module 43 indifferent ways, such as via cache-block and sub-cache-block accesses. Inaddition, the heterogeneous compute elements may comprise differentinstruction sets in certain embodiments. In other embodiments, computeelement(s) 41 may be homogeneous compute elements that have the sameinstruction sets.

Compute element(s) 41 request access to memory module 43 via bus 44.Memory controller 42 may receive such request and control assignment ofthe request to an appropriate portion of memory, such as to one of aplurality of memory modules 43 that may be implemented (only one memorymodule is illustrated for ease of discussion in FIG. 4, but a pluralityof such modules may be present in a given system, just as a plurality ofDIMMs are commonly employed for implementing main memory of computingsystems).

In the exemplary embodiment of FIG. 4, memory module 43 comprises anaddress/control channel and a plurality of data channels, shown as datachannel 0-data channel N. As shown, the address/control channel and datachannels are employed for communication between memory controller 42 andmemory module 43. In general, the address/control channel specifiesaddresses and a desired type of accesses (e.g., reads or writes) formemory access operations, and the data channels carry the correspondingdata to/from the specified addresses for performing the desired type ofaccesses. For instance, a row select command and column access commandfor memory access requests are communicated via the address/controlchannel, and data for a memory access operation (e.g., data to be readfrom memory or data to be written to memory) is carried on the datachannels.

In this exemplary embodiment, memory module 43 comprises control logic(e.g., an FPGA, ASIC, etc.) 401, as well as memory (data storage) 402.The memory 402 may be implemented by one or more memories (shown asMemory 0-Memory N), such as DRAMs (Dynamic Random Access Memory), forexample, as is commonly employed in DIMMs. In one embodiment, memorymodule (e.g., DIMM) 43 comprises eight independent data channels,wherein each of the eight independent data channels supports a sub-cachedata access. For instance, in one embodiment, each of the eightindependent data channels supports a 8-byte burst of data for acorresponding memory access operation. As an example, each of the datachannels may be implemented as 1-byte in width and employed for eachmemory access for supplying a data burst of length 8 (8 time units, suchas 8 clock cycles or 8 phases of a clock), thus resulting in an 8-byteburst of data.

For instance, in one embodiment, the 64-bit wide data path of atraditional DIMM is partitioned into eight 8-bit wide paths (i.e., datachannels 0-N of FIG. 4) that each have independent control. As anexample, in one embodiment, each of data channels 0-N is 8 bits (1-byte)wide, wherein each data access comprises a “burst” length of 8, thusresulting in each data channel 0-N carrying 8 bytes for each access.Thus, the traditional 64-byte data channel of a DIMM (such as thatdiscussed above in FIG. 2) is, in one embodiment, effectively dividedinto eight independent data channels that each support an independentaccess of 8 bytes. Accordingly, whereas a traditional data channel ofFIG. 2 provides a 64-byte data burst for a given memory access operation(read or write), an exemplary implementation of DIMM 43 according to oneembodiment effectively sub-divides the traditional DIMM data channel toprovide eight independent 8-byte data bursts for potentially supportingeight independent 8-byte access operations (read or write operations).

Further, independent memory access operations may be supported inparallel on the different data channels 0-N of FIG. 4. For instance, inthe above-mentioned exemplary embodiment in which eight data channelsare implemented, eight 8-byte data accesses can be performed in parallelon the given memory module 43. An exemplary technique for tiling theaccesses of the multiple data channels of a DIMM according to oneembodiment is described further hereafter with FIG. 7.

FIG. 5 shows a block diagram of an exemplary implementation of a memorymodule according to one embodiment of the present invention. Asdiscussed above with FIG. 4, memory module 43 includes control logic 401that receives address/control information from memory controller 42 viaan address/control channel. In this example, control logic 401 isoperable to multiplex the received address/control information intomultiple internal address/control channels, shown as address/controlchannels 501-504 in the example of FIG. 5. Each of the internaladdress/control channels 501-504 is used for supplying address/controlinformation for two data channels. For instance, tiling may be employedto enable 2 groups of DRAMs to share one control/address bus at fulldata bandwidth for both. Thus, in the illustrated example, the addressand control signals are organized on the memory module (e.g., DIMM) 43into 4 groups (e.g., groups 501-504), each serving two data channels.DRAMs are implemented for providing the data for eight data channels,shown as data channels 505 ₀-505 ₇. As shown, in this example each datachannel has a data width of 8 bits (i.e., data[7:0]), and includes a bitfor error correction code (ECC[0]).

Thus, in the exemplary implementation of FIG. 5, memory controller 42sends address/control commands to control logic 401 over address/controlchannel 500. In certain embodiments, address/control channel 500corresponds to the address/control channel of traditional DIMMs, such asaddress/control channel 204 of FIG. 2. As an example, when implementedaccording to the traditional JEDEC standard (see www.jedec.org), theaddress/control channel has a width of 32 bits, and communicatesaddress/control commands for a given memory access operation over twotime units (e.g., two clock cycles). For instance, as discussed in theexample of FIG. 3, the row select command and column access command fora single memory access operation is traditionally communicated on theaddress/control channel over two time units (i.e., two clock cycles inthat example). The address and control information that is commonlycommunicated over such an address/control channel 204 is well-known inthe art, and those of ordinary skill in the art should thereforerecognize what information may be contained in such address/controlinformation for a given memory access operation.

However, in the exemplary embodiment of FIG. 5, the address/controlcommands for a plurality of independent memory access operations arecombined so as to be communicated in a time period (e.g., over two timeunits) during which a traditional DIMM communicates address and controlcommands for a single memory access operation. In this embodiment,during the time period in which address/control channel 204traditionally communicates the address and control commands for a64-byte data access operation (e.g., read or write), address/controlchannel 500 carries information specifying the address and controlcommands for a plurality of independent sub-cache-block data accessoperations (e.g., eight 8-byte data access operations).

In the example of FIG. 5, a time multiplexed encoding scheme is used onthe address/control information communicated over channel 500. First,the address and control information (e.g., the row select and columnaccess commands) are encoded into a single command, which provides a 2:1time reduction. One exemplary technique that may be employed for suchencoding of the row select and column access commands into a singlecommand is briefly described below.

In general, DRAM accesses include a sequence of operations presented tothe DRAM via the collection of signals/commands on the address/controlchannel. These signals typically include Address/AP, Bank Address, CMD(RAS, CAS and WE), Adr/Cmd Parity, ODT and CS. A typical access sequenceincludes a bank activate (row select) command followed by a column reador column write command. Successive commands to the same row typicallyonly require a column read or column write command. Before accessinganother row on the same bank (or within a defined time limit), the rowtypically must be closed and precharged using the precharge command. Ifa single access to a row is anticipated, the precharge may be combinedwith the column access command by issuing a read or write with theauto-precharge bit set. Several of the signals are redundant orpartially used in one DRAM command or the other. For example, the BankAddress bits are the same in both row and column operations and thecolumn address does not use all of the address bits.

Standard DIMMs export the above-mentioned DRAM signals to the DIMMinterface to the memory controller. The memory controller is responsiblefor issuing the row select (or bank activate) and column access commandswith the correct sequence and timing along with the necessary prechargeoperations.

According to one embodiment, the typical row select and column accesscommands sent to the DRAM are combined into a single command sent fromthe memory controller to the DIMM. Further, according to one embodiment,this is achieved using the same total number of address and control pinsas on the standard DIMM, but the address and control pins are redefinedto carry the encoded address/control information. The resulting addresssent to the DIMM includes both the row and column addresses in a single27-bit field.

In one embodiment, some simplifications are enforced on the memorycontroller's use of commands to allow the DIMM control logic to inferthe correct sequencing of DRAM operations from the encoded DIMMcommands, using fewer total command bits. For example, in oneembodiment, a row is never left open, which implies that the DIMMcontrol logic drives the auto-precharge bit on every column accesscommand. While this precludes accessing a second column address on anopen row, the type of non-sequential access patterns for which oneembodiment of the DIMM is optimizing makes it unlikely that a subsequentaccess to a DRAM bank will be to the same row. An advantage gained fromdoing this is that no more than one DIMM command cycle is ever needed totell the DIMM control logic what sequence of operations to perform.Also, the precharge bit is not required to be sent from the memorycontroller to the DIMM. The commands sent to the DIMM in one embodimentindicate Read, Write, Refresh, Precharge and Mode Register Select. Rowactivation is inferred from a read or write command.

In one embodiment, the time between row select and column accesscommands is controlled by the DIMM control logic, rather than the memorycontroller. This allows control of the ODT signals to be moved from thememory controller into the DIMM control logic, saving these 2 signals onthe DIMM interface. In addition, multiple ranks can be supported usingfewer control bits by encoding the chip select and clock enable signalsas well, using 3 bits to carry the information normally carried by 4chip select and 2 CKE signals.

Examples of column write and read operations for both a standard DIMMand one exemplary implementation of the multi-data-channel DIMM areshown below for a 256 Mb×8 DDR2 DRAM.

-   -   Standard DIMM column write example (30 signals):        -   Row Activate command:            -   RAS,CAS,WE=011 (ACTIVATE)            -   BA<2:0>=bank number            -   A<15:0>=row address            -   CKE<1:0>=clock enables—always active other than during                initialization sequence            -   CS<3:0>=chip select—only 1 bit active            -   ODT<1:0>=On-Die Termination, controlled by MC        -   Column write command:            -   RAS, CAS, WE=100 (WRITE)            -   BA<2:0>=bank number (same as activate)            -   A<15:0>=column address (only 10 or 11 bits used)            -   CKE<1:0>=clock enables—always active other than during                initialization sequence            -   CS<3:0>=chip select—only 1 bit active            -   ODT<1:0>=On-Die Termination, controlled by MC    -   One implementation of a multi-data-channel DIMM write command        example (36 signals):        -   Write command:            -   RAS, CAS, WE=100 (WRITE, implied Activate &                Auto-Precharge)            -   BA<2:0>=bank number            -   A<26:0>=Row+column address            -   CKE<0>=clock enable—always active other than during                initialization sequence            -   CS<1:0>=chip select—encoded

In one exemplary implementation of the multi-data-channel DIMM,additional DIMM ACTL signals are obtained from a combination of unusedstrobe and DM signals, reserved and NC pins on the JEDEC DIMMdefinition. The unused strobe and DM signals are a result of the way thedata and check (ECC) bits are allocated into 8 groups of 8-bitdata+check bits instead of 9 groups of 8 bits, each group having strobeand DM bits assigned to it. There are multiple ways the standard DIMMpins could be partitioned to accomplish the same results.

Additionally, dual data rate (DDR) signaling is employed, in thisexample, to provide another factor of two bandwidth increase. Thus, thisresults in four times the address control bandwidth on channel 500 ascompared to a standard DIMM address/control channel 204 (according tothe JEDEC standard). Tiling provides an additional factor of two toallow the single address/control channel 500 to keep up with eight datachannels. An exemplary tiling scheme that may be employed is discussedfurther hereafter with FIG. 7.

The single address/control channel 500, in FIG. 5, is received bycontrol logic 401 of memory module 43, which decodes the receivedaddress/control information into information for a plurality ofdifferent DRAM address/control channels 501-504. In the illustratedexample, the address/control information received via address/controlchannel 500 is decoded by control logic 401 into correspondinginformation for four independent address/control channels, 501-504, thatare employed within memory module 43. Because address/control channel500 is used for external communication of the memory module 43 (i.e.,from memory controller 42), such address/control channel 500 may bereferred to as an external address/control channel, whereas DRAMaddress/control channels 501-504 may be referred to as internaladdress/control channels because they are used for internalcommunication within memory module 43. Each of the DRAM address/controlchannels 501-504 carries the DRAM address and control signals, such asis expected by a DRAM (e.g., according to the DRAM's specificationsheet). So, control logic 401 is operable to split the encoded, DDRaddress/control information received via address/control channel 500into four separate DRAM address/control channels 501-504 in accordancewith command timing that the individual DRAMs expect to see. Byemploying tiling in this example, each of the DRAM address/controlchannels 501-504 is used for carrying address and control informationfor two different data channels. For instance, a given DRAMaddress/control channel (e.g., channel 501) may communicate address andcontrol information for a first memory access operation to a first datachannel (e.g., data channel 505 ₀) during a first time period (e.g., twoclock cycles), and then the given DRAM address/control channel (e.g.,channel 501) may communicate address and control information for asecond memory access operation to a second data channel (e.g., datachannel 505 ₁) during a second time period.

A typical DIMM has a single data channel that is 8 bytes wide of dataand 1 byte wide of error correction code (ECC), and each memory accessreads out a burst of 8 words to result in the data channel carrying 64bytes of data plus 8 bytes of ECC for a given memory access operation.The exemplary implementation of FIG. 5 spreads the 8 bits of ECC perword across each of the plurality of data channels 505 ₀-505 ₇. So, eachdata channel 505 ₀-505 ₇ has a single ECC bit and 8 data bits. So, overa burst of 8 time units (e.g., 8 clock phases), each data channelprovides one 8-byte data word and a 1-byte ECC word that allows forsingle-bit error correction and double-bit error detection usingstandard error correction coding. Thus, together the data channels 505₀-505 ₇, over a burst of time units (e.g., 8 clock phases) provide the64-bytes of data and 8-bytes of ECC, as is typically expected in astandard DIMM. However, as discussed further herein, embodiments of thepresent invention enable each of the multiple data channels to beservicing independent memory access operations. Thus, in the example ofFIG. 5, the data paths are organized to provide independent access toeight data channels (or “lanes”) 505 ₀-505 ₇, each providing a byte-widedata path with 1 ECC bit. As discussed further herein, each DRAM isaccessed, in this exemplary embodiment, with a burst length of 8, thusproviding 8 data bytes and 1 check byte per access, per data channel.

Thus, rather than servicing a single memory access operation over atraditional single memory access time period (e.g., an 8 time unitburst), multiple data channels are employed in embodiments of thepresent invention to service a plurality of independent memory accessoperations over the same access time period. For instance, rather thancarrying 64-bytes of data for a single memory access operation over an 8time unit burst (e.g., 8 clock units or 8 clock phases), an embodimentof the multi-data channel memory module disclosed herein carries 8-bytesof data for each of a plurality of independent memory access operationsover such an 8 time unit burst. Thus, according to one embodiment, overan access time period for carrying a cache-block of data (e.g., an 8time unit burst of 64-bytes of data), the multi-data channel memorymodule carries a sub-cache-block of data for each of a plurality ofindependent memory access operations (e.g., carries 8-bytes of data foreach of eight independent memory access operations).

Turning to FIG. 6, an exemplary diagram of a DIMM 600 implementedaccording to one embodiment of the present invention is shown. DIMM 600comprises an FPGA 401A, which receives control/address commands 500A anddecodes such received control/address commands into commands for fourindependent DRAM control/address channels 501-504 (as shown in FIG. 5above). As discussed further hereafter, the integrated control element(e.g., FPGA 401A) is included on DIMM 600 to implement address andcontrol command decoding and sequencing, registering and fanout, andclock distribution. While an FPGA 401A is shown in the example of FIG.6, in certain embodiments an ASIC or other control logic for performingthe operations described may be implemented instead.

In this implementation, data channels 505 ₀-505 ₇ are each implementedwith one DRAM for providing a bit of ECC and one DRAM for providing 8bits of data. For instance, data channel 505 ₀ is formed by a first DRAM601A that provides a bit of ECC and a second DRAM 601B that provides 8bits of data (I/O 7-4 and I/O 3-0). Data channels 5051-5057 aresimilarly formed by first DRAMs 602A-608A that each provides a bit ofECC and second DRAMs 602B-608B that each provides 8 bits of data, asshown. The DRAMs thus provide eight, independent data channels 610 ₀-610₇, which correspond to data channels 0-N in the example of FIG. 4.

In the example of FIG. 6, DIMM 600 is a packaged module that can beconnected to a computer system in place of a standard DIMM module. Forinstance, DIMM 600 in this example has a standard 240-pin connector 609,as is traditionally employed for DIMMs of the prior art. In certainembodiments, an interface is provided for programming FPGA 401A. Ofcourse, in certain implementations, a persistent image may be storedlocally on DIMM 600 and loaded to FPGA 401A, rather than programmingFPGA 401A from an external source. Alternatively, in certain embodimentsan ASIC or other control logic other than an FPGA may be implemented onDIMM 600 for performing the operations described herein for such controllogic. However, in the example of FIG. 6, an additional connector, suchas side connector 611, is provided to enable an external source toconnect to FPGA 401A for programming such FPGA 401A for performing thecontrol logic operations described herein. Thus, in the illustratedexample of FIG. 6, a sideband connector 611 is used to provide a pathfor FPGA programming and voltages to the DIMM 600. This connector 611 isunused on the motherboard if a standard JEDEC DIMM is used. Further, asmentioned above, the connector 611 may be eliminated if an ASIC is usedfor the control logic on the DIMM 600 instead of an FPGA 401A. Asanother example, in certain embodiments the connector 611 may beeliminated by combining use of certain signals available via connector609 to perform the dual purpose of programming FPGA 401A in addition totheir normal use.

The exemplary embodiment of DIMM 600 in FIG. 6 maintains socket levelcompatibility with standard DDR2 RDIMM (240-pin socket), organized as 64data bits, 8 check bits (ECC) and a single set of address, control andclock/strobe signals per JEDEC spec. Further, the exemplary embodimentof FIG. 6 may be implemented to fit within the mechanical and thermalenvelope of a standard DDR2 RDIMM. This exemplary embodiment of FIG. 6maps all signals to the same memory controller pins as for a standardDIMM solution. Additionally, this exemplary embodiment includessufficient ECC bits for error correction of single bit errors and errordetection of double bit errors. Further, the exemplary embodiment ofFIG. 6 may be implemented using commodity DRAMs, which provides a costsavings for the module.

As discussed above, to improve data channel bandwidth, tiling may beemployed. FIG. 7 shows an example of a tiling technique employedaccording to one embodiment of the present invention. FIG. 7 shows clockcycles 701 of a reference clock signal, wherein the illustrated exampleshows 20 clock cycles numbered 1-20. Clock phases 702 is also shown,wherein for each clock cycle the clock has a low phase (“L”) and a highphase (“H”), as is well known. An external address/control channel, suchas channel 500 of FIG. 5, is shown as encoded address/control channel703. As discussed above with FIG. 6, preferably the address and controlcommands are communicated over the external address/control channel 703according to a time multiplexed encoding scheme. Thus, FIG. 7 shows anexample of address/control commands received by control logic 401 frommemory controller 42 via address/control channel 500.

Also, in this example, four internal DRAM address/control channels areshown as channels 704, 707, 710, and 713, which correspond to theinternal address/control channels 501-504 of FIG. 5. As discussed abovewith FIG. 5, control logic 401 decodes the address and control commandsreceived via encoded address/control channel 703 (channel 500 of FIG. 5)to produce the address and control commands for each of the fourinternal address/control channels 704, 707, 710, and 713 (channels501-504 of FIG. 5), as discussed further hereafter.

As discussed in the examples of FIGS. 5 and 6 above, each of theinternal address/control channels 704, 707, 710, and 713 (channels501-504 of FIG. 5) may support address and control commands for twodifferent data channels. Thus, in FIG. 7 a first data channel 705 and asecond data channel 706 carry data as specified by the address andcontrol commands supplied by first internal address/control channel 704.Thus, first internal address/control channel 704 may correspond tointernal address/control channel 501 of FIG. 5, and data channels 705and 706 may correspond to data channels 505 ₀-505 ₁ of FIG. 5.

Also, in FIG. 7 a third data channel 708 and a fourth data channel 709carry data as specified by the address and control commands supplied bysecond internal address/control channel 707. Thus, second internaladdress/control channel 707 may correspond to internal address/controlchannel 502 of FIG. 5, and data channels 708 and 709 may correspond todata channels 505 ₂-505 ₃ of FIG. 5. Similarly, a fifth data channel 711and a sixth data channel 712 carry data as specified by the address andcontrol commands supplied by third internal address/control channel 710(e.g., the third internal address/control channel 710 may correspond tointernal address/control channel 503 of FIG. 5, and data channels 711and 712 may correspond to data channels 505 ₄-505 ₅ of FIG. 5). Finally,in FIG. 7, a seventh data channel 714 and an eighth data channel 715carry data as specified by the address and control commands supplied byfourth internal address/control channel 713 (e.g., fourth internaladdress/control channel 713 may correspond to internal address/controlchannel 504 of FIG. 5, and data channels 714 and 715 may correspond todata channels 505 ₆-505 ₇ of FIG. 5).

In the illustrated example of FIG. 7, in the low phase of clock cycle 0,encoded address/control command 716 is received by control logic 401 (ofFIG. 5) via address/control channel 703. Control logic 401 decodes thereceived command 716 to produce the address and control commands forinternal address/control channel 704 (e.g., internal address/controlchannel 501 of FIG. 5). In this example, the address and control command716 received by control logic 401 can be decoded to produce both a rowselect command and a column access command for a memory accessoperation. Thus, as a result of such decoding, control logic 401 placesrow select command 720 for a first memory access operation on internaladdress/control channel 704 in clock cycle 1, and places column accesscommand 721 for the first memory access operation on internaladdress/control channel 704 in clock cycle 2.

After a predefined delay (the DRAM's data access delay), data channel705 carries the data “burst” for the first memory access operation. Forinstance, beginning in the high phase of clock cycle 9 and ending in thelow phase of clock cycle 13, data burst 722 carries the data for thefirst memory access operation. In this exemplary implementation, databurst 722 carries 8-bytes of data for the first memory access operation.For instance, data channel 705 is implemented as an 8-bit (1-byte) widechannel, wherein each memory access comprises a “burst” length of 8 timeunits (e.g., clock phases), thus resulting in the data channel carrying8 bytes of data for each access. For instance, each of the 8 blocks ofburst 722 (labeled 0/0/0-0/0/7) may be a 1-byte block of data, thusresulting in burst 722 containing 8 bytes of data for the first memoryaccess operation (read or write to/from the specified address).

Continuing with the illustrated example of FIG. 7, in the high phase ofclock cycle 0, encoded address/control command 717 is received bycontrol logic 401 (of FIG. 5) via address/control channel 703. Controllogic 401 decodes the received command 717 to produce the address andcontrol command for internal address/control channel 710 (e.g., internaladdress/control channel 503 of FIG. 5). In this example, the address andcontrol command 717 received by control logic 401 can be decoded toproduce both a row select command and a column access command for amemory access operation. Thus, as a result of such decoding, controllogic 401 places row select command 731 for a second memory accessoperation on internal address/control channel 710 in clock cycle 2, andplaces column access command 732 for the second memory access operationon internal address/control channel 710 in clock cycle 3.

After a predefined delay (the DRAM's data access delay), data channel711 carries the data “burst” for the second memory access operation. Forinstance, beginning in the high phase of clock cycle 10 and ending inthe low phase of clock cycle 14, data burst 733 carries the data for thesecond memory access operation. In this exemplary implementation, databurst 733 carries 8-bytes of data for the second memory accessoperation. For instance, data channel 711 is implemented as an 8-bit(1-byte) wide channel, wherein each memory access comprises a “burst”length of 8 time units (e.g., clock phases), thus resulting in the datachannel carrying 8 bytes of data for each access.

Continuing further with the illustrated example of FIG. 7, in the lowphase of clock cycle 1, encoded address/control command 718 is receivedby control logic 401 (of FIG. 5) via address/control channel 703.Control logic 401 decodes the received information 718 to produce rowselect command 737 and column access command 738 for a third memoryaccess operation, which are placed on internal address/control channel707 in clock cycles 2 and 3, respectively. In response, beginning in thehigh phase of clock cycle 10 and ending in the low phase of clock cycle14, data burst 739 carries the data for the third memory accessoperation on data channel 708 in a manner similar to that discussedabove for data bursts 722 and 733.

In the high phase of clock cycle 1, encoded address/control command 719is received by control logic 401 (of FIG. 5) via address/control channel703. Control logic 401 decodes the received command 719 to produce rowselect command 743 and column access command 744 for a fourth memoryaccess operation, which are placed on internal address/control channel713 in clock cycles 3 and 4, respectively. In response, beginning in thehigh phase of clock cycle 11 and ending in the low phase of clock cycle15, data burst 745 carries the data for the fourth memory accessoperation on data channel 714 in a manner similar to that discussedabove for data bursts 722 and 733.

In the low phase of clock cycle 2, encoded address/control command 750is received by control logic 401 (of FIG. 5) via address/control channel703. Control logic 401 decodes the received command 750 to produce rowselect command 723 and column access command 724 for a fifth memoryaccess operation, which are placed on internal address/control channel704 in clock cycles 3 and 4, respectively. In response, beginning in thehigh phase of clock cycle 11 and ending in the low phase of clock cycle15, data burst 725 carries the data for the fifth memory accessoperation on data channel 706 in a manner similar to that discussedabove for data bursts 722 and 733.

In the high phase of clock cycle 2, encoded address/control command 751is received by control logic 401 (of FIG. 5) via address/control channel703. Control logic 401 decodes the received command 751 to produce rowselect command 734 and column access command 735 for a sixth memoryaccess operation, which are placed on internal address/control channel710 in clock cycles 4 and 5, respectively. In response, beginning in thehigh phase of clock cycle 12 and ending in the low phase of clock cycle16, data burst 736 carries the data for the sixth memory accessoperation on data channel 712 in a manner similar to that discussedabove for data bursts 722 and 733.

In the low phase of clock cycle 3, encoded address/control command 752is received by control logic 401 (of FIG. 5) via address/control channel703. Control logic 401 decodes the received command 752 to produce rowselect command 740 and column access command 741 for a seventh memoryaccess operation, which are placed on internal address/control channel707 in clock cycles 4 and 5, respectively. In response, beginning in thehigh phase of clock cycle 12 and ending in the low phase of clock cycle16, data burst 742 carries the data for the seventh memory accessoperation on data channel 709 in a manner similar to that discussedabove for data bursts 722 and 733.

In the high phase of clock cycle 3, encoded address/control command 753is received by control logic 401 (of FIG. 5) via address/control channel703. Control logic 401 decodes the received command 753 to produce rowselect command 746 and column access command 747 for an eighth memoryaccess operation, which are placed on internal address/control channel713 in clock cycles 5 and 6, respectively. In response, beginning in thehigh phase of clock cycle 13 and ending in the low phase of clock cycle17, data burst 748 carries the data for the eighth memory accessoperation on data channel 715 in a manner similar to that discussedabove for data bursts 722 and 733.

Operation may continue in a similar manner, as illustrated in FIG. 7.The exemplary tiling technique of FIG. 7 allows for the address/controlchannel 703, as well as the internal address/control channels 704, 707,710, and 713, to be used to maintain high bandwidth utilization of thedata channels 705-706, 708-709, 711-712, and 714-715, as illustrated inFIG. 7. This tiling enables the same memory bandwidth for non-sequentialaccess of 8-byte quantities of data to be supported as for traditionalsequential 64-byte access. For instance, as shown, the exemplary tilingtechnique of FIG. 7 allows for data burst 727 to occur immediatelyfollowing data burst 722 on data channel 705 for servicing anothermemory access operation, thus maintaining high bandwidth utilization ondata channel 705. Similarly, as illustrated in FIG. 7, the exemplarytiling technique allows for data burst 730 to occur immediatelyfollowing data burst 725 on data channel 706 for servicing anothermemory access operation, thus maintaining high bandwidth utilization ondata channel 706.

Thus, in the above example of FIGS. 5-7, a memory module is providedthat includes eight independent 8-bit wide data channels. In operation,for a respective independent memory access operation being serviced byone of the eight data channels, the data channel carries data in a burstof 8 time units (e.g., 8 clock phases), thereby carrying 8 bytes of datafor servicing the respective independent memory access operation. Thus,over a time period during which a cache-block of data (e.g., 64 bytes ofdata) is carried by the data channels, each data channel carries anindependent 8-byte sub-cache-block of data. While a specific example ofeight data channels that are each 8-bits in width carry data in a burstof 8 time units, it should be recognized that this is merely anillustrative example, and in other embodiments a different number ofindependent data channels each having different width and/or carryingdata in a burst of more or less than 8 time units may be employed. Thus,those of ordinary skill in the art should recognize that the conceptspresented herein are not limited to the specific architecture described,but may likewise be employed for other memory module architectures thatemploy a plurality of independent data channels, wherein any number oftwo or more data channels may be employed, each data channel may haveany desired width, and/or each data channel may carry data for anindependent memory access over any time frame as may be desired for agiven system.

It should be recognized that embodiments of the multi-data channelmemory module may, in some implementations, be employed across multipleDRAM ranks. For instance, as is well known in the art, a singleaddress/control channel, such as address/control channel 500 of FIG. 5,may be employed for use by multiple ranks.

FIG. 8 shows an exemplary system 80 in which multi-data channel memorymodules according to embodiments of the present invention may beimplemented. An embodiment of the exemplary system 80 is describedfurther in concurrently-filed U.S. patent application Ser. No.12/186,344 entitled “MEMORY INTERLEAVE FOR HETEROGENEOUS COMPUTING,” thedisclosure of which is incorporated herein by reference. For instance,as discussed further below, embodiments of the multi-data channel memorymodules as described herein may be implemented as the DIMM modules 805₀-805 ₁, 806 ₀-806 ₁, and 807 ₀-807 ₁ of memory subsystem 83 of system80.

In exemplary system 80, a processing subsystem 81 and a memory subsystem83 are provided. In this exemplary embodiment, processing subsystem 81comprises compute elements 21A and 21B. Compute element 21A iscache-block oriented and issues to a memory interleave system a physicaladdress for a cache-block memory access request, while compute element21B is sub-cache-block oriented and issues to the memory interleavesystem a virtual address for a sub-cache-block access request. Asdiscussed hereafter, in this example, the memory interleave systemcomprises a host interface 802 that receives requests issued by computeelement 21A, and the memory interleave system comprises a memoryinterface 803 that receives requests issued by heterogeneous computeelement 21B.

In this exemplary implementation, the storage elements associated witheach memory controller 22 ₀-22 _(N) comprise a pair of DIMMs. Forinstance, a first pair of DIMMs 805 ₀-805 ₁ is associated with memorycontroller 22 ₀, a second pair of DIMMs 806 ₀-806 ₁ is associated withmemory controller 22 ₁, and a third pair of DIMMs 807 ₀-807 ₁ isassociated with memory controller 22 _(N). In one embodiment, there are8 memory controllers implemented, but a different number may beimplemented in other embodiments. The DIMMs may each comprise amulti-data channel memory module, such as the exemplary embodimentsdescribed above with FIGS. 2-7.

Further details regarding exemplary system 80, including a memoryinterleaving scheme that may be employed therein, are described inconcurrently-filed U.S. patent application Ser. No. 12/186,344 entitled“MEMORY INTERLEAVE FOR HETEROGENEOUS COMPUTING,” the disclosure of whichis incorporated herein by reference. While system 80 provides oneexample of a system in which multi-data channel memory modules may beimplemented, embodiments of the multi-data channel memory modulesdisclosed herein are not limited in application to this exemplary system80, but may likewise be employed in any other system in which suchmulti-data channel memory modules may be desired.

In certain embodiments, the multi-data channel memory module may beutilized for supporting cache-block memory accesses, as well assupporting sub-cache-block data accesses. In certain embodiments, uponreceiving a cache-block access request, the eight data channels 505₀-505 ₇ (of FIG. 5) may be reserved (to place any sub-cache-block accessrequests received thereafter “on hold” until the eight data channels areused for satisfying the cache-block access request), and the eight datachannels may then be used to fully, in one burst, satisfy the 64-bytecache-block access request, in a manner similar to traditional DIMMs.

In other embodiments, no such reservation is employed for cache-blockaccess requests, but instead the cache-block access request may behandled by the eight data channels 505 ₀-505 ₇ (of FIG. 5) along with anintermingling of any sub-cache-block access requests that might bepresent at that time, wherein the cache-block access may be satisfied bythe data channels within a window of time, and the 64 bytes of thecache-block access returned by the data channels within the window oftime may be bundled by logic (e.g., memory controller 42) into therequested 64 byte cache block of data. Thus, rather than supplying thesame address and request type (e.g., either a read or write) to alleight data channels simultaneously for satisfying a cache-block accessrequest, in certain embodiments, such address and request type for thecache-block access may in a first instance be supplied to a portion ofthe eight data channels (which each returns their respective portion ofthe requested cache block) and in a later instance (within a window oftime) a further portion of the eight data channels may be supplied theaddress and request type in order to return the remaining portion of therequested cache block. The two portions of the cache block may thenbundled together (e.g., by memory controller 42) to form the requestedcache block of data. In other words, rather than satisfying acache-block access in a single burst of data, in certain embodimentsportions of the cache-block of data may be returned over a plurality ofbursts (e.g., with independent sub-cache-block bursts of dataintermingled therewith), and the appropriate portions may be bundledtogether to form a congruent burst of cache-block data.

Thus, in certain embodiments, cache-block (e.g., 64-byte) accesses maybe intermixed with sub-cache-block (e.g., 8-byte) accesses, and each8-byte data channel 505 ₀-505 ₇ (of FIG. 5) of the memory module isscheduled independently to support the intermixing. Thus, a cache-blockaccess may not necessarily be performed using all eight data channelsfor returning the entire cache-block in a single, congruent burst in themanner mentioned above, but instead, at a given time some of the eight8-byte data channels may be used for performing a sub-cache-block accesswhile some others of the eight 8-byte data channels are used for thecache-block access. Thus, the cache-block access may be returned withina window of time by the data channels, wherein a controller 42 bundlesthe returned data into the requested cache-block.

In certain embodiments, the multi-data channel memory module may beconfigurable into either of at least two modes of operation. Forinstance, in one embodiment, the multi-data channel memory module may bestatically or dynamically configurable (e.g., through programming ofFPGA 401A of FIG. 6) to operate as a traditional DIMM in which themultiple data channels are used together as a single data channel forservicing received memory access requests via cache-block bursts ofdata, such as discussed above with FIG. 2. And, the multi-data channelmemory module may be statically or dynamically configurable (e.g.,through programming of FPGA 401A of FIG. 6 or selecting an operatingmode through a register access) to operate in the manner discussed abovewith FIGS. 4-7 such that the multiple data channels are operable to eachservice independent memory access requests via sub-cache-block bursts ofdata. In certain embodiments, the control logic (e.g., FPGA 401A) mayinclude both a configuration for operating as a traditional DIMM (inwhich the multiple data channels are used together as a single datachannel for servicing received memory access requests via cache-blockbursts of data) and a configuration for operating as amulti-data-channel module as discussed above, and a command may bereceived (e.g., from a register) to select which of the configurationsshould be active at any given time for servicing memory access requests.In this way, an executable (e.g., software application) or portionthereof that may benefit from cache-block oriented memory accesses candynamically configure the multi-data channel memory module for operatingin a traditional manner, whereas an executable (e.g., softwareapplication) or portion thereof that may benefit from sub-cache-blockoriented memory accesses can dynamically configure the multi-datachannel memory module for employing its multiple data channels for eachservicing independent memory access requests via sub-cache-block burstsof data such as discussed above with FIGS. 4-7.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

What is claimed is:
 1. An apparatus comprising: a processor configuredto provide memory commands for memory access operations; a memorycontroller coupled to the processor; and a memory comprising: aplurality of memory elements; and control logic coupled to the memorycontroller via a control channel, the control logic configured to decodecontrol information from the control channel and to selectively providea respective portion of decoded control information to at least one ofthe plurality of memory elements via a respective internal controlchannel of a plurality of internal control channels, independently ofthe other internal control channels of the plurality of internal controlchannels.
 2. The apparatus of claim 1, wherein the control logic isconfigured to control data transfer over at least one data channel of aplurality of data channels, each data channel coupled to a respectiveone of the plurality of memory elements.
 3. The apparatus of claim 2,wherein each data channel provides data independently of the other datachannels.
 4. An apparatus comprising: a processor configured to providememory commands for memory access operations; and a memory controllercoupled to the processor; and a memory comprising: a plurality of memoryelements; and control logic coupled to the memory controller via acontrol channel, the control logic is further configured to select aninternal control channel of a plurality of internal control channelsbased on decoded control information, wherein each internal controlchannel provides at least a portion of the decoded control information,independently of the other internal control channel of the plurality ofinternal control channels, to a respective memory element of theplurality of memory elements.
 5. The apparatus of claim 4, wherein thecontrol logic is configured to control a time between the memory accessoperations.
 6. The apparatus of claim 5, wherein the memory accessoperations comprise row select commands and column access commands. 7.The apparatus of claim 4, wherein the control logic is implemented on afield-programmable gate array (FPGA) or ASIC.
 8. An apparatuscomprising: a processor comprising an instruction set including a memoryaccess request; a memory controller coupled to the processor andconfigured to receive the memory access request and to provideaddress/control information responsive to the memory access request; anda memory comprising a plurality of memory elements and control logic,wherein the control logic is coupled to the memory controller via anexternal address/control channel, the control logic configured toreceive and decode the address/control information and to provide thedecoded address/control information to at least one of the plurality ofmemory elements to control data transfer over multiple data channels,wherein each memory element is coupled to the control logic via arespective internal address/control channel of a plurality of internaladdress/control channels, wherein each internal address/control channelprovides respective decoded internal address/control information,independently of the other internal address/control channels.
 9. Theapparatus of claim 8, wherein the instruction set comprises independentmemory access requests for the plurality of memory elements, theindependent memory access requests including the memory access request.10. The apparatus of claim 9, wherein the control logic is configured todecode the address/control information into component address/controlinformation, wherein each component address/control informationcorresponds to one of the independent memory access requests.
 11. Theapparatus of claim 8, wherein the control logic is configured to decodethe address/control information based on a time multiplexed encodingscheme.
 12. The apparatus of claim 8, wherein each of the respectiveinternal address/control channels of the plurality of internaladdress/control channels provides the respective decoded internaladdress/control information to each respective memory element of theplurality of memory elements.
 13. The apparatus of claim 12, wherein theaddress/control information comprises the respective decoded internaladdress/control information, and wherein the respective decoded internaladdress/control information is provided in parallel over the respectiveinternal address/control channel of the plurality of internaladdress/control channels.
 14. The apparatus of claim 8, wherein a firstdata channel of the multiple data channels is configured to transferdata based on a first memory access request, and wherein a second datachannel of the multiple data channels is configured to transfer databased on a second memory access request.
 15. The apparatus of claim 8,wherein the apparatus corresponds to a field-programmable gate array(FPGA).
 16. The apparatus of claim 8, wherein each memory element of theplurality of memory elements supports sub-cache data access operations.17. The apparatus of claim 8, wherein each memory element of theplurality of memory elements comprise at least one dynamic random accessmemory (DRAM).